pypy support: fix a cpyext incompatibility in chunkers/reader, adjust test expectations, see #1755 - #9976
Draft
ThomasWaldmann wants to merge 3 commits into
Draft
pypy support: fix a cpyext incompatibility in chunkers/reader, adjust test expectations, see #1755#9976ThomasWaldmann wants to merge 3 commits into
ThomasWaldmann wants to merge 3 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9976 +/- ##
==========================================
- Coverage 86.90% 86.88% -0.02%
==========================================
Files 101 101
Lines 17863 17864 +1
Branches 2708 2708
==========================================
- Hits 15523 15521 -2
- Misses 1636 1637 +1
- Partials 704 706 +2 ☔ View full report in Codecov by Harness. |
This was referenced Jul 30, 2026
ThomasWaldmann
marked this pull request as draft
July 30, 2026 20:19
ThomasWaldmann
force-pushed
the
try-pypy
branch
from
August 17, 2026 17:37
666ecd5 to
739f608
Compare
ThomasWaldmann
changed the title
ThomasWaldmann
force-pushed
the
try-pypy
branch
from
August 17, 2026 17:53
739f608 to
0cfa4cf
Compare
pypy's cpyext memoryview does not support slice assignment (mv[a:b] = src): the statement works in interpreted code, but raises TypeError when it is compiled into a C extension, like in this .pyx module. That broke about 500 tests on pypy - basically every operation that writes an archive. Copying via memcpy through Cython typed memoryviews avoids creating the slice objects and is a wash on CPython (1.689 vs 1.680 GB/s, best of 6, chunking a 2 GiB file through the block reader path). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
borg works on pypy3.11: tested with a py3.11 nightly build (8ed7cc691813, PyPy 8.0.0-alpha0) on macOS/arm64, where the full test suite passes (2534 passed, 1 xfailed) and CPython is unaffected. The pypy incompatibilities found in the first round (see borgbackup#1755 and the previous version of this branch) are all fixed in pypy now, so their workarounds are gone again: fcntl.F_FULLFSYNC (pypy/pypy#5543), hmac.digest with memoryview/bytearray (pypy/pypy#5544), os.link(follow_symlinks=False) (pypy/pypy#5545). The chunk-data memoryview leak (pypy/pypy#5546) is fixed, too: chunking 21.5 GB without releasing the memoryviews now peaks at 252 MB instead of growing linearly with the data volume. What remains is not pypy bugs: - platformflags: add is_pypy. - item_test: xfail test_unknown_property - setting undeclared attributes on cdef class instances is not blocked under cpyext. - msgpack_test: pypy only has the pure-python msgpack, so expect it to be slow there. - lock_cmds_test: tolerate the pure-python msgpack warning on stderr, and use sys.executable instead of "python3" (robustness, not pypy-specific). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ThomasWaldmann
force-pushed
the
try-pypy
branch
from
August 18, 2026 02:53
0cfa4cf to
6cb7205
Compare
…rgbackup#1755 Without os.readv (win32, pypy) we read via os.read and copy into the scan buffer. os.read allocates a buffer of the requested size, no matter how few bytes it then returns, so requesting the whole free scan buffer (up to 8MiB for the default max chunk size) is expensive for small files: one allocation of that size per read call, several calls per file. Backing up 20000 small files (82.6MiB of data) requested 468GiB in 60002 os.read calls. Capping the request at 256KiB - same call count, same data - cuts the time spent in os.read from 15.1s to 1.3s and the total runtime from 27.6s to 12.3s on pypy (which also zeroes the allocation). On CPython without readv it is 4.04s -> 2.42s, close to the 2.36s of the readv path. Chunking a 2GiB file is unaffected (the big reads there return what they ask for). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With this PR, borg works on pypy3.11, tested with a py3.11 nightly build
(
pypy-c-jit-187982-8ed7cc691813, PyPy 8.0.0-alpha0 / Python 3.11.15) on macOS/arm64:the full test suite passes there (2534 passed, 384 skipped, 1 xfailed), and CPython is
unaffected (full suite verified, no regressions). A nightly is required - the pypy fixes borg
needs are not in a release yet.
Changes
chunkers/reader: memcpy instead of memoryview slice assignment. This is what currentlykeeps borg from running on pypy at all:
mv[a:b] = srcraisesTypeErrorwhen the statementis compiled into a C extension - it works in interpreted code, and item assignment works -
because pypy's cpyext memoryview lacks the assignment slot (pypy/pypy#5564). Without this,
506 of ~2900 tests fail, i.e. everything that writes an archive. Copying via
memcpy()through Cython typed memoryviews avoids creating the slice objects and is a wash on CPython:
1.689 vs 1.680 GB/s (best of 6, chunking a 2 GiB file through the block reader path).
chunkers/reader: limit the read size in the no-readv fallback. Not pypy-specific, butthat is where it hurts most: without
os.readv(win32, pypy - pypy/pypy#5565) we read withos.read, which allocates a buffer of the requested size no matter how few bytes it thenreturns (pypy also zeroes it, pypy/pypy#5566). Asking for the whole free scan buffer (up to
8 MiB) therefore costs MBs of allocation per small file. Backing up 20000 small files
(82.6 MiB of data) requested 468 GiB in 60002 read calls; capping the request at 256 KiB -
same call count, same data - cuts the time in
os.readfrom 15.1 s to 1.3 s:pypy support proper, all of it about behaviour that is not a pypy bug:
platformflags: addis_pypy.item_test: xfailtest_unknown_property- setting undeclared attributes on cdef classinstances is not blocked under cpyext.
msgpack_test: pypy only has the pure-python msgpack, expect it to be slow there.lock_cmds_test: tolerate the pure-python msgpack warning on stderr, and usesys.executableinstead of"python3"(robustness, not pypy-specific).Performance
pypy remains slower than CPython for borg's workload and this PR does not change that:
borg createof a 2 GiB file is 4.8 s vs 10.8 s, 20000 small files 2.4 s vs 12.3 s (it was27 s before the read-size cap above). Profiling says the rest is the cpyext tax, spread over
all of borg's C extensions rather than one hot spot - a trivial C ext call is 304 ns on pypy
vs 45 ns on CPython (a pure python call is 1 ns vs 50 ns), a 4 KiB bytes object created in
C 3.0 us vs 0.6 us, AES-OCB on a 2 KiB chunk 20.8 us vs 2.0 us, lz4 12.2 us vs 1.0 us,
ChunkIndex set+get 20.2 us vs 1.8 us. Where borg is plain Python, pypy is faster
(
borg create -n, walk and stat only: 0.71 s vs 0.43 s including interpreter startup).So: compatibility, not speed.
See #1755. The CI job running the suite on a pypy nightly came from #10142 and #10143.